An experimental characterization of workers'' behavior and accuracy in crowdsourced tasks

Crowdsourcing systems are evolving into a powerful tool of choice to deal with repetitive or lengthy human-based tasks. Prominent among those is Amazon Mechanical Turk, in which Human Intelligence Tasks, are posted by requesters, and afterwards selected and executed by subscribed (human) workers in the platform. Many times these HITs serve for research purposes. In this context, a very important question is how reliable the results obtained through these platforms are, in view of the limited control a requester has on the workers'' actions. Various control techniques are currently proposed but they are not free from shortcomings, and their use must be accompanied by a deeper understanding of the workers'' behavior. In this work, we attempt to interpret the workers'' behavior and reliability level in the absence of control techniques. To do so, we perform a series of experiments with 600 distinct MTurk workers, specifically designed to elicit the worker''s level of dedication to a task, according to the task''s nature and difficulty. We show that the time required by a worker to carry out a task correlates with its difficulty, and also with the quality of the outcome. We find that there are different types of workers. While some of them are willing to invest a significant amount of time to arrive at the correct answer, at the same time we observe a significant fraction of workers that reply with a wrong answer. For the latter, the difficulty of the task and the very short time they took to reply suggest that they, intentionally, did not even attempt to solve the task. © 2021 Christoforou et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

Applying the dynamics of evolution to achieve reliability in master-worker computing

We consider Internet-based master-worker task computations, such as SETI@home, where a master process sends tasks, across the Internet, to worker processes; workers execute and report back some result. However, these workers are not trustworthy, and it might be at their best interest to report incorrect results. In such master-worker computations, the behavior and the best interest of the workers might change over time. We model such computations using evolutionary dynamics, and we study the conditions under which the master can reliably obtain task results. In particular, we develop and analyze an algorithmic mechanism based on reinforcement learning to provide workers with the necessary incentives to eventually become truthful. Our analysis identifies the conditions under which truthful behavior can be ensured and bounds the expected convergence time to that behavior. The analysis is complemented with illustrative simulations.This work is supported by the Cyprus Research Promotion Foundation grant TΠE/ΠΛHPO/0609(BE)/05, the National Science Foundation (CCF-0937829, CCF-1114930), Comunidad de Madrid grants S2009TIC-1692 and MODELICO-CM, Spanish PRODIEVO and RESINEE grants and MICINN grant EC2011-29688-C02-01, and National Natural Science Foundation of China grant 61020106002.Publicad

Achieving Reliability in Master-worker Computing via Evolutionary Dynamics

The proceeding at: 18th International Conference on Parallel and Distributed Computing, Euro-Par 2012), took place 2012, August 27-31, in Rhodes Island, Greece.This work considers Internet-based task computations in which a master process assigns tasks, over the Internet, to rational workers and collect their responses. The objective is for the master to obtain the correct task outcomes. For this purpose we formulate and study the dynamics of evolution of Internet-based master-worker computations through reinforcement learning.This work is supported by the Cyprus Research Promo-tion Foundation grant TΠE/ΠΛHPO/0609(BE)/05, NSF grants CCF-0937829, CCF-1114930, Comunidad de Madrid grant S2009TIC-1692, Spanish MOSAICO and RESINEE grants and MICINN grant TEC2011-29688-C02-01, and National Natural Science Foundation of China grant 61020106002.Publicad

Investigation of the influence of rail hardness on the wear of rail and wheel materials under dry conditions (ICRI wear mapping project)

Some railway managers and practitioners fear that introducing premium rail materials will have a detrimental effect on the wheels of trains that use the line. A review of relevant investigations across all scales in the laboratory, and in the field has been carried out. This showed that, as rail hardness increases, its wear, and overall system wear reduces. Wheel wear does increase with increasing rail hardness, but only for wheels running on rails that are softer than them. Similar trends were observed in all studies, so it seems that the fears were unfounded. While the wear trends appear well characterised some issues have been identified. One relates to the varying work hardening capability of wheel and rail materials. Often only bulk hardness is quoted, but work hardening can increase material surface hardness by up to 2.5 times and make materials that were initially softer, harder than the opposing material. Another related issue is test length. It is essential that enough cycles are applied such that the materials reach steady state wear, i.e., the point at which work hardening has reached its limit. In previous work it is not always clear that steady state wear has been reached. Some gaps have been identified in the current knowledge base, the largest of which is the failure to determine which mechanisms lead to the wear trends seen. Analysis of recent work on different clad layers on rail discs and premium rail materials allowed some of these gaps to be addressed. Results indicated that opposing wheel material hardened to the same level independent of rail hardness. Wheel wear is therefore stress driven under the conditions used, and dictated by the wheel material properties only. At higher slip levels relationships become less clear, but here temperature and therefore hot hardness is most influential and is as yet uncharacterised

SBV Regularity for Genuinely Nonlinear, Strictly Hyperbolic Systems of Conservation Laws in one space dimension

We prove that if $t \mapsto u(t) \in \mathrm {BV}(\R)$ is the entropy solution to a $N \times N$ strictly hyperbolic system of conservation laws with genuinely nonlinear characteristic fields $$u_t + f(u)_x = 0,$$ then up to a countable set of times $\{t_n\}_{n \in \mathbb N}$ the function $u(t)$ is in $\mathrm {SBV}$, i.e. its distributional derivative $u_x$ is a measure with no Cantorian part. The proof is based on the decomposition of $u_x(t)$ into waves belonging to the characteristic families $$u(t) = \sum_{i=1}^N v_i(t) \tilde r_i(t), \quad v_i(t) \in \mathcal M(\R), \ \tilde r_i(t) \in \mathrm R^N,$$ and the balance of the continuous/jump part of the measures $v_i$ in regions bounded by characteristics. To this aim, a new interaction measure \mu_{i,\jump} is introduced, controlling the creation of atoms in the measure $v_i(t)$. The main argument of the proof is that for all $t$ where the Cantorian part of $v_i$ is not 0, either the Glimm functional has a downward jump, or there is a cancellation of waves or the measure $\mu_{i,\mathrm{jump}}$ is positive

Twin disc assessment of wear regime transitions and rolling contact fatigue in R400HT – E8 pairs

Twin disc tests were carried out to evaluate the wear resistance and Rolling Contact Fatigue (RCF) of premium R400HT rail samples in contact with E8 wheel samples. The wear rate and friction coefficient were correlated with the frictional work expended at the contact interface (the Tgamma approach). Accelerated RCF tests were also carried out on the premium R400HT rail and the results were compared to those obtained for standard R260 rail. The wear rates of rail samples were consistently lower than those reported in the literature for other contacting pairs in which the rail material studied is softer than R400HT. Also, the energy needed for the transition from the moderate to severe wear regime significantly increased for the hardened rail. Fatigue cracks were shallower for R400HT when compared with standard rail material. Hardened rails also showed lower mean spacing between fatigue cracks. This new information can be used to improve wear simulations of wheels and rails by using more realistic wear equations

The effect of organelle discovery upon sub-cellular protein localisation.

Prediction of protein sub-cellular localisation by employing quantitative mass spectrometry experiments is an expanding field. Several methods have led to the assignment of proteins to specific subcellular localisations by partial separation of organelles across a fractionation scheme coupled with computational analysis. Methods developed to analyse organelle data have largely employed supervised machine learning algorithms to map unannotated abundance profiles to known protein–organelle associations. Such approaches are likely to make association errors if organelle-related groupings present in experimental output are not included in data used to create a protein–organelle classifier. Currently, there is no automated way to detect organelle-specific clusters within such datasets. In order to address the above issues we adapted a phenotype discovery algorithm, originally created to filter image-based output for RNAi screens, to identify putative subcellular groupings in organelle proteomics experiments. We were able to mine datasets to a deeper level and extract interesting phenotype clusters for more comprehensive evaluation in an unbiased fashion upon application of this approach. Organelle-related protein clusters were identified beyond those sufficiently annotated for use as training data. Furthermore, we propose avenues for the incorporation of observations made into general practice for the classification of protein–organelle membership from quantitative MS experiments. Biological significance Protein sub-cellular localisation plays an important role in molecular interactions, signalling and transport mechanisms. The prediction of protein localisation by quantitative mass-spectrometry (MS) proteomics is a growing field and an important endeavour in improving protein annotation. Several such approaches use gradient-based separation of cellular organelle content to measure relative protein abundance across distinct gradient fractions. The distribution profiles are commonly mapped in silico to known protein–organelle associations via supervised machine learning algorithms, to create classifiers that associate unannotated proteins to specific organelles. These strategies are prone to error, however, if organelle-related groupings present in experimental output are not represented, for example owing to the lack of existing annotation, when creating the protein–organelle mapping. Here, the application of a phenotype discovery approach to LOPIT gradient-based MS data identifies candidate organelle phenotypes for further evaluation in an unbiased fashion. Software implementation and usage guidelines are provided for application to wider protein–organelle association experiments. In the wider context, semi-supervised organelle discovery is discussed as a paradigm with which to generate new protein annotations from MS-based organelle proteomics experiments. This article is part of a Special Issue entitled: New Horizons and Applications for Proteomics [EuPA 2012]